[OpenMP] Fix runtimes default build #149871

Meinersbur · 2025-07-21T18:57:39Z

The default build of openmp (cmake -S <llvm-project>/runtimes -DLLVM_ENABLE_RUNTIMES=openmp) current fails with

CMake Error at /home/meinersbur/src/llvm/flangrt/_src/cmake/Modules/GetClangResourceDir.cmake:17 (string):
  string sub-command REGEX, mode MATCH needs at least 5 arguments total to
  command.
Call Stack (most recent call first):
  /home/meinersbur/src/llvm/flangrt/_src/openmp/CMakeLists.txt:126 (get_clang_resource_dir)

The reason is that because it is not a bootstrapping-build, the clang resource dir that it intends to write files such as omp-tools.h into, is unavailable. Using the Clang resource dir for writing files is conceptually broken, as that dir might be located in /usr/lib/clang/<version>/. Writing to it is only intended in bootstrapping builds where Clang is built alongside openmp.

This patch unifies the identification of being in a bootstrapping built. The same LLVM_TREE_AVAILABLE definition is going to be used in #137828. No reason for each runtime to define its own.

runtimes/CMakeLists.txt

Co-authored-by: Joseph Huber <[email protected]>

The default build of openmp (`cmake -S <llvm-project>/runtimes -DLLVM_ENABLE_RUNTIMES=openmp`) current fails with ``` CMake Error at /home/meinersbur/src/llvm/flangrt/_src/cmake/Modules/GetClangResourceDir.cmake:17 (string): string sub-command REGEX, mode MATCH needs at least 5 arguments total to command. Call Stack (most recent call first): /home/meinersbur/src/llvm/flangrt/_src/openmp/CMakeLists.txt:126 (get_clang_resource_dir) ``` The reason is that because it is not a bootstrapping-build, the clang resource dir that it intends to write files such as `omp-tools.h` into, is unavailable. Using the Clang resource dir for writing files is conceptually broken, as that dir might be located in `/usr/lib/clang/<version>/`. Writing to it is only intended in bootstrapping builds where Clang is built alongside openmp. This patch unifies the identification of being in a bootstrapping built. The same `LLVM_TREE_AVAILABLE` definition is going to be used in llvm#137828. No reason for each runtime to define its own.

madhur13490 · 2025-07-29T04:27:29Z

Hi @Meinersbur

This commit has broken 3-stage AArch64 builds. Without this build, #include <omp.h> and -fopenmp were enough to compile a simple OpenMP code, but since this has changed the location of omp.h, clang can't compile now.

We use 3-stage builds quite often; I appreciate the recommendation or a fix here soon.

clang version 22.0.0git (https://github.com/llvm/llvm-project.git b39160ddfb625bd57f9113471b15cda748151a10)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /local/home/work/llvm-project/build-3-stage/build_Release_LTO-PGO-BOLT_main/bin
Found candidate GCC installation: /usr/lib/gcc/aarch64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/aarch64-linux-gnu/12
Selected GCC installation: /usr/lib/gcc/aarch64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/local/home/work/llvm-project/build-3-stage/build_Release_LTO-PGO-BOLT_main/bin/clang-22" -cc1 -triple aarch64-unknown-linux-gnu -emit-obj -dumpdir /local/home/work/o.out- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name openmp.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=non-leaf -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -enable-tlsdesc -target-cpu generic -target-feature +v8a -target-feature +fp-armv8 -target-feature +neon -target-abi aapcs -debugger-tuning=gdb -fdebug-compilation-dir=/local/home/madhura/llvm-project/build-3-stage/build_Release_LTO-PGO-BOLT_main/bin -v -fcoverage-compilation-dir=/local/home/madhura/llvm-project/build-3-stage/build_Release_LTO-PGO-BOLT_main/bin -resource-dir /local/home/work/llvm-project/build-3-stage/build_Release_LTO-PGO-BOLT_main/lib/clang/22 -internal-isystem /local/home/work/llvm-project/build-3-stage/build_Release_LTO-PGO-BOLT_main/lib/clang/22/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/aarch64-linux-gnu/12/../../../../aarch64-linux-gnu/include -internal-externc-isystem /usr/include/aarch64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -ferror-limit 19 -fmessage-length=126 -fopenmp -fno-signed-char -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcolor-diagnostics -target-feature +outline-atomics -target-feature -fmv -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/openmp-d06f23.o -x c /local/home/work/openmp.c
clang -cc1 version 22.0.0git based upon LLVM 22.0.0git default target aarch64-unknown-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/aarch64-linux-gnu/12/../../../../aarch64-linux-gnu/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /local/home/work/llvm-project/build-3-stage/build_Release_LTO-PGO-BOLT_main/lib/clang/22/include
 /usr/local/include
 /usr/include/aarch64-linux-gnu
 /usr/include
End of search list.
/local/home/work/openmp.c:2:10: fatal error: 'omp.h' file not found
    2 | #include <omp.h>
      |          ^~~~~~~
1 error generated.

Here is the simple code:

#include <stdio.h>
#include <omp.h>

int main(int argc, char** argv){
    printf("Hello from process: %d\n", omp_get_thread_num());

    return 0;
}

I believe as this commit has changed the location of omp.h, it has also broken many apps including SPEC 2017 as all the apps include omp.h with just #include <omp.h>.

jprotze · 2025-07-29T07:22:35Z

@madhur13490 can you provide information about the three stages (i.e., relevant cmake flags for all stages like llvm-projects/runtime, flags to customize openmp build/install paths)?
Additionall, information about the location of omp.h before this patch and with this patch might be helpful to understand the problem.

jprotze · 2025-07-29T07:33:17Z

openmp/runtime/src/CMakeLists.txt

@@ -11,7 +11,7 @@
 include(ExtendPath)

 # The generated headers will be placed in clang's resource directory if present.
-if(OPENMP_STANDALONE_BUILD OR NOT LLVM_RUNTIMES_BUILD)
+if(NOT LLVM_TREE_AVAILABLE)


When building openmp as llvm project, LLVM_TREE_AVAILABLE is undefined, so the wrong branch will be used

I only considered the standalone build when LLVM_TREE_AVAILABLE being undefined just means means the LLVM tree is indeed not available. Will create a fix.

Note that that build mode is deprecated. #124014.

madhur13490 · 2025-07-29T08:23:50Z

@madhur13490 can you provide information about the three stages (i.e., relevant cmake flags for all stages like llvm-projects/runtime, flags to customize openmp build/install paths)? Additionall, information about the location of omp.h before this patch and with this patch might be helpful to understand the problem.

omp.h is moved from lib/clang/22/include/omp.h to include/omp.h.

We use standard 3-stage LTO-PGO-BOLT builds, which primarily use https://github.com/llvm/llvm-project/blob/main/clang/cmake/caches/BOLT-PGO.cmake. You can find top-level community doc at https://llvm.org/docs/AdvancedBuilds.html

jprotze · 2025-07-29T09:21:37Z

omp.h is moved from lib/clang/22/include/omp.h to include/omp.h.

This fits to my in code comment above.

We use standard 3-stage LTO-PGO-BOLT builds, which primarily use https://github.com/llvm/llvm-project/blob/main/clang/cmake/caches/BOLT-PGO.cmake. You can find top-level community doc at https://llvm.org/docs/AdvancedBuilds.html

These documents don't include OpenMP in their builds.
In the caches directors, only VectorEngines.cmake builds OpenMP via LLVM_ENABLE_RUNTIMES.
The important question: do you build OpenMP via LLVM_ENABLE_PROJECTS or LLVM_ENABLE_RUNTIMES?

madhur13490 · 2025-07-29T09:48:03Z

omp.h is moved from lib/clang/22/include/omp.h to include/omp.h.

This fits to my in code comment above.

We use standard 3-stage LTO-PGO-BOLT builds, which primarily use https://github.com/llvm/llvm-project/blob/main/clang/cmake/caches/BOLT-PGO.cmake. You can find top-level community doc at https://llvm.org/docs/AdvancedBuilds.html

These documents don't include OpenMP in their builds. In the caches directors, only VectorEngines.cmake builds OpenMP via LLVM_ENABLE_RUNTIMES. The important question: do you build OpenMP via LLVM_ENABLE_PROJECTS or LLVM_ENABLE_RUNTIMES?

via LLVM_ENABLE_PROJECTS.

Meinersbur · 2025-07-29T09:49:48Z

omp.h is moved from lib/clang/22/include/omp.h to include/omp.h.

I assume lib/clang/22/include/omp.h is in the Clang build tree and include/omp.h is in the OpenMP build tree.

The openmp build must not write outside its own build tree. Doing so causes countless problems such as conflicting versions of omp.h when building multiple openmp builds in different directories (such as Release/Debug), or the Clang build directory being read-only.

"/local/home/work/llvm-project/build-3-stage/build_Release_LTO-PGO-BOLT_main/bin/clang-22" ...

This looks like you are invoking clang from the build directory. Either switch to a bootstrapping-build of openmp, or install both Clang and openmp into the same CMAKE_INSTALL_PREFIX and launch Clang from there.

Meinersbur · 2025-07-29T10:19:01Z

via LLVM_ENABLE_PROJECTS.

In this case the Clang build tree and the openmp build tree are identical. Fixed in #151117.

Note that this build mode is deprecated since #136314. Consider moving to a runtimes build. According to the warning message added in #136314, that build mode should already have been removed.

madhur13490 · 2025-08-01T04:59:03Z

via LLVM_ENABLE_PROJECTS.

In this case the Clang build tree and the openmp build tree are identical. Fixed in #151117.

Note that this build mode is deprecated since #136314. Consider moving to a runtimes build. According to the warning message added in #136314, that build mode should already have been removed.

Thanks for all the help. We switched to ENABLE_RUNTIMES in our CI and we're back to green.

Set LLVM_TREE_AVAILABLE when not defined after #149871. In particular, the LLVM build tree is obviously available with `add_subdirectory(openmp)` from the LLVM build tree itself. Note that this build mode is deprecated since #136314.

Set LLVM_TREE_AVAILABLE when not defined after llvm#149871. In particular, the LLVM build tree is obviously available with `add_subdirectory(openmp)` from the LLVM build tree itself. Note that this build mode is deprecated since llvm#136314.

[OpenMP] default build fix

dbfe5a8

Meinersbur mentioned this pull request Jul 21, 2025

[OpenMP] Remove standalone build mode #149878

Draft

Meinersbur changed the title ~~[OpenMP] Fix default build~~ [OpenMP] Fix runtimes default build Jul 21, 2025

Meinersbur mentioned this pull request Jul 21, 2025

[Flang][OpenMP] Move builtin .mod generation into runtimes #137828

Open

Meinersbur marked this pull request as ready for review July 21, 2025 23:24

Meinersbur requested a review from a team as a code owner July 21, 2025 23:24

llvmbot added cmake Build system in general and CMake in particular openmp:libomp OpenMP host runtime openmp:libomptarget OpenMP offload runtime labels Jul 21, 2025

Meinersbur requested review from petrhosek, jhuber6 and jprotze July 22, 2025 15:32

jhuber6 approved these changes Jul 22, 2025

View reviewed changes

runtimes/CMakeLists.txt Outdated Show resolved Hide resolved

Meinersbur requested a review from hahnjo July 22, 2025 15:34

Apply change suggested by @jhuber6

9fce817

Co-authored-by: Joseph Huber <[email protected]>

Meinersbur merged commit 799f28d into main Jul 25, 2025
76 checks passed

Meinersbur deleted the users/meinersbur/openmp_fix-runtimes-build branch July 25, 2025 23:56

jprotze reviewed Jul 29, 2025

View reviewed changes

Meinersbur mentioned this pull request Jul 29, 2025

[OpenMP] Fix LLVM_ENABLE_PROJECTS=openmp build #151117

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[OpenMP] Fix runtimes default build #149871

[OpenMP] Fix runtimes default build #149871

Uh oh!

Meinersbur commented Jul 21, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

madhur13490 commented Jul 29, 2025 •

edited

Loading

Uh oh!

jprotze commented Jul 29, 2025

Uh oh!

jprotze Jul 29, 2025

Uh oh!

Meinersbur Jul 29, 2025 •

edited

Loading

Uh oh!

madhur13490 commented Jul 29, 2025

Uh oh!

jprotze commented Jul 29, 2025

Uh oh!

madhur13490 commented Jul 29, 2025

Uh oh!

Meinersbur commented Jul 29, 2025 •

edited

Loading

Uh oh!

Meinersbur commented Jul 29, 2025 •

edited

Loading

Uh oh!

madhur13490 commented Aug 1, 2025

Uh oh!

Uh oh!

[OpenMP] Fix runtimes default build #149871

[OpenMP] Fix runtimes default build #149871

Uh oh!

Conversation

Meinersbur commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

madhur13490 commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jprotze commented Jul 29, 2025

Uh oh!

jprotze Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Meinersbur Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

madhur13490 commented Jul 29, 2025

Uh oh!

jprotze commented Jul 29, 2025

Uh oh!

madhur13490 commented Jul 29, 2025

Uh oh!

Meinersbur commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Meinersbur commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

madhur13490 commented Aug 1, 2025

Uh oh!

Uh oh!

Meinersbur commented Jul 21, 2025 •

edited

Loading

madhur13490 commented Jul 29, 2025 •

edited

Loading

Meinersbur Jul 29, 2025 •

edited

Loading

Meinersbur commented Jul 29, 2025 •

edited

Loading

Meinersbur commented Jul 29, 2025 •

edited

Loading